Multifunction fingerprint sensor having optical sensing capability

ABSTRACT

A fingerprint sensor device includes a touch panel with an integrated touch sensor module is provided. The integrated touch sensor module includes sensing circuitry to generate a sensor signal responsive to detecting a contact input associated with a fingerprint. The sensing circuitry includes a fingerprint sensor to detect the contact input and generate a signal indicative of an image of the fingerprint, and a biometric sensor to generate a signal indicative of a biometric marker different form the fingerprint. The generated sensor signal includes the signal indicative of the image of the fingerprint and the signal indicative of the biometric marker different from the fingerprint. The sensing circuitry includes processing circuitry communicatively coupled to the sensing circuitry to process the generated sensor signal to determine whether the contact input associated with the fingerprint belongs to a live finger.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/US2016/038445, filed on Jun. 20, 2016 and entitled “MULTIFUNCTIONFINGERPRINT SENSOR HAVING OPTICAL SENSING CAPABILITY,” which claims thebenefit of priority of U.S. Provisional Patent Application No.62/181,718, filed on Jun. 18, 2015, and entitled “MULTIFUNCTIONFINGERPRINT SENSOR AND PACKAGING,” both of which are incorporated byreference in its entirety as part of the disclosure of this patentdocument.

TECHNICAL FIELD

This patent document generally relates to fingerprint recognition forsecurely accessing an electronic device that includes mobile andwearable devices.

BACKGROUND

Electronic devices including portable or mobile computing devices, suchas laptops, tablets, smartphones, and gaming systems can employ userauthentication mechanisms to protect personal data and preventunauthorized access. User authentication on an electronic device can becarried out through one or multiple forms of biometric identifiers,which can be used alone or in addition to conventional passwordauthentication methods. A popular form of biometric identifiers is aperson's fingerprint pattern. A fingerprint sensor can be built into theelectronic device to read a user's fingerprint pattern so that thedevice can only be unlocked by an authorized user of the device throughauthentication of the authorized user's fingerprint pattern.

SUMMARY

The examples of implementations described in this patent documentprovide fingerprint sensor designs that use optical sensors for sensingfingerprints or a combination of optical sensors and another type offingerprint sensors such as capacitive sensors to sense fingerprints.The described fingerprint sensor designs can be used in various devices,systems or applications, and can be configured to be particularlysuitable for mobile applications, and various wearable or portabledevices.

In one aspect, an electronic device with touching sensing andfingerprint sensing capabilities is provided to include a touch screenthat provides touch sensing operations; a top transparent layer formedover the touch screen as an interface for being touched by a user forthe touch sensing operations; and an optical sensor module located belowthe touch screen to receive light that is returned from the toptransparent layer and transmits through the touch screen. The opticalsensor module includes an optical detector array of photodetectorspositioned to receive at least a portion of the returned light to detecta fingerprint. In one implementation, the optical sensor module includesan array of optical collimators located between the touch screen and theoptical detector array to direct the received portion of the returnedlight to the photodetectors through the optical collimators. In anotherimplementation, the touch screen includes a fingerprint sensing zone fora user to touch for fingerprint sensing to generate the returned lightreceived by the optical detector array for detecting a fingerprint; afirst optical sensing zone and a second optical sensing zone; and theoptical module includes (1) a first additional optical detector locatedon a first side of the optical detector array to receive a portion ofthe returned light from the first optical sensing zone, and (2) a secondadditional optical detector located on a second opposite side of theoptical detector array to receive a portion of the returned light fromthe second optical sensing zone. The first and second additional opticaldetectors produce detector signals indicating whether the returned lightis reflected from a finger of a live person. In yet anotherimplementation, the touch screen includes display pixels that producelight for displaying images and for illuminating a user's finger intouch with the top transparent layer to produce the returned light; andthe touch screen includes a fingerprint sensing zone for a user to touchfor fingerprint sensing to receive illumination light from one or moredisplay pixels that are located to generate illumination light to thefingerprint sensing zone in a way that the illumination light undergoestotal optical reflection at a top surface of the top transparent layerto direct the totally reflected light to the optical detector array fordetecting a fingerprint.

In another aspect, a fingerprint sensor device includes a touch panelwith an integrated touch sensor module. The integrated touch sensormodule includes sensing circuitry to generate a sensor signal responsiveto detecting a contact input associated with a fingerprint. The sensingcircuitry includes a fingerprint sensor to detect the contact input andgenerate a signal indicative of an image of the fingerprint, and abiometric sensor to generate a signal indicative of a biometric markerdifferent form the fingerprint. The generated sensor signal includes thesignal indicative of the image of the fingerprint and the signalindicative of the biometric marker different from the fingerprint. Thesensing circuitry includes processing circuitry communicatively coupledto the sensing circuitry to process the generated sensor signal todetermine whether the contact input associated with the fingerprintbelongs to a live finger. In implementations, the fingerprint sensor orthe biometric sensor of the sensing circuitry includes an opticalsensor.

In another aspect, an electronic device is provided to include a centralprocessor; a touch panel in communication with the central processor;and a fingerprint sensor device integrated into the touch panel and incommunication with the central processor. The fingerprint sensor deviceincludes a sensing circuitry to generate a sensor signal responsive todetecting a contact input associated with a fingerprint. The sensingcircuitry includes a fingerprint sensor to detect the contact input andgenerate a signal indicative of an image of the fingerprint, and abiometric sensor to generate a signal indicative of an identification ofa biometric marker different from the fingerprint. The generated sensorsignal includes the signal indicative of the image of the fingerprintand the signal indicative of the biometric marker different from thefingerprint. Processing circuitry is communicatively coupled to thesensing circuitry to process the generated sensor signal to determinewhether the contact input associated with the fingerprint belongs to alive finger. In implementations, the fingerprint sensor or the biometricsensor of the sensing circuitry includes an optical sensor.

In another aspect, a method is provided for detecting a live fingerduring a fingerprint scan and includes detecting, by a fingerprintsensor, a contact input associated with a source of a fingerprint;generating an image signal from the fingerprint sensor responsive to thedetected contact input, wherein the generated image signal from thefingerprint sensor is indicative of one or more images of thefingerprint; generating, by a biometric sensor, a biometric markerdetection signal indicative of a biometric marker different from thefingerprint; and processing, by processing circuitry, the generatedimage signal and the biometric marker detection signal to determinewhether the detected contact and the associated one or more fingerprintimages are from a live finger. In implementations, the fingerprintsensor or the biometric sensor of the sensing circuitry includes anoptical sensor.

In yet another aspect, a method is provided of finger scanning andincludes initiating sensor detection to activate a sensor moduleincluding a light source and a light detector; controlling the lightsource to modulate light beams emitted by the light source to carrymodulated signal information including amplitude, phase shift, frequencychange, or a combination; acquiring an optical signal in response to theemitted modulated light beams; demodulating the acquired optical signal;and processing the demodulated signal to generate a fingerprint imageand obtain a biometric marker different from the fingerprint.

The above and other aspects, their implementations and applications aredescribed in greater detail in the drawings, the description and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a visible optical fingerprint sensor packagein a mobile device and includes FIGS. 1A, 1B and 1C showing threedifferent views of features in connection with the visible opticalfingerprint sensor package.

FIGS. 2A and 2B are block diagrams of an exemplary fingerprint sensingmodule in a visible package for installing in a mobile device, such asthe mobile device of FIG. 1.

FIG. 3 is a block diagram of an exemplary fingerprint sensing module ina visible package for heartbeat sensing.

FIG. 4 is a block diagram of an exemplary fingerprint sensing module ina visible package for blood flow speed sensing.

FIG. 5 is a graph showing exemplary signals from blood flow speedsensing.

FIG. 6 is a block diagram showing an example of an invisible opticalfingerprint sensor package in a mobile device and FIGS. 6A, 6B and 6Cshowing three different views of features in connection with theinvisible optical fingerprint sensor package.

FIGS. 7A and 7B are block diagrams of an exemplary fingerprint sensingmodule in an invisible package for installing in a mobile device, suchas the mobile device of FIG. 6.

FIG. 8 is a block diagram of an exemplary fingerprint sensing module inan invisible package for heartbeat sensing.

FIG. 9 is a block diagram of an exemplary fingerprint sensing module inan invisible package for blood flow speed sensing.

FIGS. 10A and 10B are cross-sectional and top-down views of an exemplaryfingerprint sensor module implementing a total reflection fingerprintsensing technique.

FIGS. 11A and 11B are cross-sectional and top-down views of an exemplaryfingerprint sensor module implementing total reflection fingerprintdetection scanning.

FIGS. 12A, 12B, and 12C represent an exemplary fingerprint sensor moduleinstalled in a mobile device and implementing total reflectionfingerprint detection scanning.

FIG. 13 is an image representing a result of a total reflectionfingerprint detection scanning.

FIGS. 14A, 14B, and 14C represent another exemplary fingerprint sensormodule installed in a mobile device and implementing total reflectionfingerprint detection.

FIGS. 15A and 15B represent another exemplary fingerprint sensor moduleinstalled in a mobile device and implementing total reflectionfingerprint sensing.

FIG. 16 is a process flow diagram showing an exemplary process 1600 forfingerprint detection.

FIGS. 17, 18A and 18B are diagrams showing an example of totalreflection touch sensing-refractive index matching technique.

FIGS. 19A, 19B, and 19C are diagrams showing an example of a totalreflection touch sensing-refractive index matching technique.

FIG. 20A is a block diagram of an exemplary fingerprint sensor deviceimplementing self-capacitive sensing with active sensor pixel andamplification.

FIG. 20B shows an exemplary sensor pixel.

FIG. 20C shows the circuit equivalent of the sensor pixel.

FIG. 21A shows an example of a fingerprint sensor device thatincorporates a capacitive sensor in addition to an optical sensor foreach sensor pixel in capturing fingerprint information.

FIG. 21B illustrates another example of a fingerprint sensor device thatstructurally integrates an optical sensor and a capacitive sensor ineach hybrid sensor pixel in a spatially overlap configuration in anarray of sensor pixels to reduce the footprint of each hybrid sensingpixel.

FIG. 22 is a top-down view of an exemplary hybrid fingerprint sensordevice incorporating both an optical sensor and a capacitive sensor ineach hybrid sensing pixel.

FIG. 23A illustrates a circuit diagram for an exemplary hybridfingerprint sensing element or pixel having both capacitive sensing andoptical sensing functions for fingerprints.

FIG. 23B illustrates a circuit diagram for another exemplary hybridfingerprint sensing element or pixel.

FIG. 23C illustrates a circuit diagram of an exemplary hybridfingerprint sensing element or pixel for performing parallel detectionof sensor signals from the photodetector and the capacitive sensorplate.

FIGS. 24A, 24B, 24C and 24D show process flow diagrams of an exemplaryprocess for performing fingerprint sensing by a hybrid fingerprintsensor that incorporates optical and capacitive sensors.

FIGS. 25A and 25B illustrate an example of such a design for a mobilephone, a tablet or other devices where the optical sensor module isoutside the display screen assembly 10 as a separate structure.

DETAILED DESCRIPTION

Fingerprint sensing is a useful use authentication tool in mobileapplications and other applications that use, provide or require secureaccess. For example, fingerprint sensing can be used to provide secureaccess to a mobile device, an electronic device or system, or anelectronic portal to one or more systems or databases, and can be usedto secure financial transactions including online purchases and otherdigital information. It is desirable to include robust and reliablefingerprint sensors features suitable for mobile devices or portabledevices while providing a small footprint for such fingerprint sensorswith a thin structure to fit into the highly limited space in mobile orportable devices and other compact devices. In some applications, it isalso desirable to include a protective cover to protect such afingerprint sensor from various contaminants and to provide a userinterface for touch sensing and certain user operations.

The technology disclosed in this patent document can be used toimplement different types of fingerprint sensors either individually orin combinations, including capacitive fingerprint sensors and opticalfingerprint sensors. In capacitive fingerprint sensors, the sensing isbased on measuring the capacitance between the sensing electrode and afinger surface due to their capacitive coupling and variation in thecapacitive coupling strength due to the surface topology of thefingerprint pattern including locations and shapes of fingerprint ridgesand valleys. A protective cover can be placed over the capacitive sensorpixels to protect the capacitive fingerprint sensor. As this protectivecover becomes thicker, the electrical field sensed by each capacitivesensor pixel disperses quickly in space, and the sensor signal strengthreceived at each sensor pixel also reduces significantly with theincrease in thickness of the protective cover. In addition, theincreased protective cover thickness can lead to an undesirable steepreduction in the spatial resolution of the sensor. In some devices, whenthe protective cover thickness exceeds a certain threshold (e.g., 300μm), it can become difficult for such capacitive sensors to provide adesired high spatial resolution in sensing fingerprint patterns and toreliably resolve a sensed fingerprint pattern with an acceptablefidelity.

As part of the disclosed technology in this patent document, afingerprint sensor can also be implemented for optically capturingfingerprints. The thickness of an optical fingerprint module for opticalfingerprint sensing tends to cause the optical fingerprint sensor to betoo thick for certain applications or under certain technical orengineering constraints and the thickness of the optical fingerprintsensor tends to make integrating to a mobile phone device or a compactdevice difficult due to limitations or requirements in technology andengineering.

In one aspect, the disclosed technology provides a fingerprint sensordesign for an ultrathin optical fingerprint sensor for integration intoa mobile or compact device. In another aspect of the disclosedtechnology, a hybrid fingerprint sensor is provided to include bothoptical and capacitive sensors in each sensing pixel of a pixelatedsensor array. The optical sensors can be packaged as photodiode arraysat a suitable location, including a detector location underneath of thedisplay (e.g., or edges or a peripheral region of the display) or alocation outside of but adjacent to the display. A window or a partiallytransparent coating can be used with or in the detectors. Examples ofimplementations of the disclosed technology can be used to introduce anoptical technology for sensing finger properties including fingerprintdetection. The optical fingerprint sensing technology can be used for awide range of devices that have a display structure. The opticalfingerprint sensing technology can be packaged in a discrete device insome designs.

Visible Optical Fingerprint Sensor Package

A visible optical fingerprint sensor package can be used to dispose oneor more photodiode arrays of an optical fingerprint sensor on at one ormore display side or edge positions outside the display screen area tomake the presence of the optical fingerprint sensor “visible” on thedevice in the context that the optical fingerprint sensor module ispartially or entirely outside the display screen area. A window or apartially transparent coating or cover can be used for the detectorarrays in such a visual optical fingerprint sensor package.

FIG. 1 shows an example of a visible optical fingerprint sensor packagein a mobile device with three different views of features in connectionwith the visible optical fingerprint sensor package. FIG. 1 includesthree different views in FIGS. 1A (top view from the display side of themobile device), 1B (side view along the line B-B′ in FIG. 1A to showdifferent layers in the fingerprint sensor) and 1C (showing some detailsof example of optical detector arrangement of the fingerprint sensor).Referring to FIG. 1A, this example of a mobile device includes a displayassembly 10 with an integrated finger property sensor component. Thedisplay assembly 10 includes a display screen that can be implemented bya LCD screen, an OLED screen or other display screen designs. Thedisplay screen in the display assembly 10 an be a touch sensitive screento provide a touch sensing user interface in operating the mobiledevice. Referring to FIG. 1B, the display assembly 10 includes anenhanced cover glass 50 on the top and other display layers 54 for thedisplay screen disposed below the enhanced cover glass 50. As shown inFIG. 1A, reference number 12 indicates one example of a position ofanother sensor and two or more other sensors can be placed on the mobiledevice beyond the fingerprint sensor. The mobile device can also includeoptional user input mechanisms such as side buttons 14, 16 prepared forthe smart terminals. Reference number 21 represents the fingerprintsensor zone, where the fingerprint sensor is located. Reference number23 represents a detector array associated with the fingerprint sensor insome fingerprint sensor implementations (FIG. 1C). The fingerprintsensor zone 21 is located outside the display screen of the displayassembly 10 at a selected location to allow the optical detectors withinfingerprint sensor zone 21 for capturing fingerprints to be able toreceive reflected or scattered light from a user's finger for thefingerprint sensing operations. As shown by the insert drawing in FIG.1B, disposed above the optical detector array 23 are receiving optics 24(e.g., optical lenses or optical collimators), which are the opticaldetectors for detecting light reflected off of a target, such as afinger. The receiving optics 24 above the photodetector array 23 can bein various configurations, e.g., it may include one or more lenses insome designs, optical collimators without lenses, or combining lensesand collimators. A back board 25 with integrated circuitry is disposedbelow the detector array 23 as shown in FIG. 1B. In someimplementations, the detector array 23 can be integrated into thebackboard 25. The backboard 25 can be disposed over a flexible printedcircuit (FPC) 27. For the specific design shown in FIGS. 1A and 1B wherethe optical detector array 23 for the optical fingerprint sensing isplaced next to one end of the display screen structure 54, a supportglass 56 may be disposed at the other side of the optical fingerprintsensor module under the cover glass 50 to strengthen the mechanicalstrength of the overall assembly at the optical fingerprint sensormodule area and to improve the structural integrity of the overallpackaging. For devices that use the cover glass 50 with a high strengthmaterial or with a sufficient thick glass, this support glass 56 may beeliminated.

Referring back to FIG. 1A, located near the fingerprint sensor zone 21within the display screen of the display assembly 10 is a specifieddisplay zone 29 for fingerprint detection on which a user places a userfinger for the fingerprint detection based on optical fingerprintsensing by the underneath fingerprint sensor. This specified displayzone 29 for fingerprint detection is part of the display screen and thusis used for both displaying images and for receiving fingerprintpatterns from a user's finger. The specified display zone 29 forfingerprint detection as part of the display screen is located adjacentto the fingerprint sensor zone 21 located outside the display screen ofthe display assembly 10 so that the reflected or scattered light fromthe illuminated portion of the finger pressed against the display zone29 can enter the fingerprint sensor zone 21 to reach the opticaldetector array 23 for optical sensing operations. As shown in FIG. 1B, alight path window 31 is disposed above the receiving optics and near thespecified display zone 29 within the device display for fingerprintdetection. The detector array 23 includes multiple detector elementsarranged in different detector zones including a fingerprint detectorzone 33 located in the central area of the detector array 23 forfingerprint and fingerprint property detection and one or moreadditional detector zones 41 and 43 for other optical sensing functions,where the additional detector zones 41 and 43, as illustrated in FIG.1C, may be environment and blood flow detector zones 41 and 43 forenvironment and blood flow speed detection. In addition, referencenumbers 37 and 39 represent additional specified zones that may beplaced over an enhanced cover glass 50 for other optical sensingoperations in some implementations, such as blood flow speed detection.In some implementations, each of the additional sensing zones 37 and 39may include a patch of display pixels in the display screen structure 54that can operate to produce desired illumination for the additionaloptical sensing, e.g., emitting red light or light of a desired spectralrange to illuminate a user's finger for sensing blood flow speed orglucose level. This design represents an example where thelight-emitting touch screen (e.g., an OLED touch screen) includes afingerprint sensing zone 29 for a user to touch for fingerprint sensingto generate the returned light received by the optical detector arrayfor detecting a fingerprint, a first optical sensing zone 37 and asecond optical sensing zone 39 on two opposite sides of the zone 29 toprovide for additional optical sensing beyond the fingerprint sensing.

As shown in FIG. 1B, this particular optical fingerprint sensor designis different from some other fingerprint sensor designs using a separatefingerprint sensor structure from the display screen with a physicaldemarcation between the display screen and the fingerprint sensor (e.g.,a button like structure in an opening of the top glass cover in somemobile phone designs) on the surface of the mobile device. Under theillustrated design in FIG. 1B the fingerprint sensor zone 21 and theassociated optical detector sensor module 23 for detecting fingerprintsensing and other optical signals are located under the top cover glassor layer 50 so that the top surface of the cover glass or layer 50serves as the top surface of the mobile device as a contiguous anduniform glass surface across both the display screen of the displayassembly 10 (containing the specified display zone 29 for fingerprintdetection near the edge of the display screen) and the fingerprintsensor zone 21 and the associated optical detector sensor module 23.This design for integrating optical fingerprint sensing and the touchsensitive display screen under a common and uniform surface providesbenefits, including improved device integration, enhanced devicepackaging, enhanced device resistance to failure and wear and tear, andenhanced user experience. This feature is also present in otherimplementations of the disclosed technology in this document. However,in some implementations of the optical sensing of fingerprints and othersensing operations, the optical sensor module may be packaged in adiscrete device configuration in which the optical sensor module is adistinct structure that has a structural border or demarcation with thedisplay screen, e.g., a button like fingerprint sensor structure in anopening of the top glass cover in some mobile phone designs based on alloptical sensing or a hybrid sensing with both capacitive sensing andoptical sensing.

A visible package of a fingerprint sensor can be used in any deviceswith display or similar light sources. The optical sensing in FIG. 1 isbased on illumination of a user's finger. Different illuminationmechanisms may be used. For example, in some implementations, thedisplay elements of the display screen that are located within theseveral zones 29, 37, 39 in FIG. 1A within the display screen foroptical sensing can be used to illuminate the user's finger. The lightfrom such display elements of the display screen is used to form part ofthe displayed images on the display screen as part of the mobile deviceoperation and the light, after transmitting through the top layer of thescreen, will illuminate the user's finger to cause reflected orscattered light from the illuminated finger to allow optical sensing bythe detector array 23 and other detectors 41, 43, 37 and 39. A suitabledisplay screen for implementing the disclosed optical sensor technologycan be based on various display technologies or configurations,including, a display screen having light emitting display pixels withoutusing backlight where each individual pixel generates light for forminga display image on the screen such as an organic light emitting diode(OLED) display screens or electroluminescent display screens. Thedisclosed optical sensor technology may also be adapted for use withother display screens, such as LCD display screens which use one or moreillumination light sources (e.g., LEDs) to produce illumination light ina backlighting or edge lighting configuration to illuminate the LCDdisplay pixels which filter and modulate the illumination light at eachLCD pixel to display the images.

When the display illumination light is used to illuminate the user'sfinger for optical sensing of the user's fingerprint or other biometricparameters, the light from the display, which can be directly emitted bydisplay pixels (e.g., OLED display pixels) or can be optically filteredby the display pixels (e.g., LED display pixels based on backlighting oredge lighting designs), contains different colors, e.g., red (575 nm-660nm), green (490 nm-575 nm) and blue (410 nm-490 nm) light. For opticalsensing other than fingerprints, such as the blood flow speed orheartbeat rate, the optical wavelength for the light that illuminatesthe user's finger may be selected at certain optical wavelengths, e.g.,in the red spectra for sensing a user's blood to obtain the heartbeatrate, the oxygen level, the glucose level and others. In an OLED displayscreen, each display color pixel includes at least OLED pixels at threedifferent colors and the red light from red color OLED pixels can be tomeasure the blood information of a user.

Several integrated detector arrays 23, 33, 41, 43 (e.g. photo diodes)can be used to detect the light scattered from the finger tissues anddetect the light in the environment. The detector arrays 23, 33, 41, 43can be placed and packaged close to the specified display zones 29, 37,39 to enhance the light detection efficiency. Depending on theapplications and specific device designs, the positions of the specifieddisplay zones shown in FIG. 1 may be modified from the exemplarylocations as shown in FIG. 1A and be placed at other suitable locationson the display assembly 10.

In various implementations, one or more other light sources may be usedto produce light to illuminate a user's finger for optically sensing thefingerprints or other biometric parameters of the user. Such light foroptical sensing is different and separate from the display illuminationlight that is either emitted by display pixels (e.g., OLED displaypixels) or is directed from illumination light sources to the displaypixels (e.g., LED display pixels based on backlighting or edge lightingdesigns). The one or more light sources for optical sensing may beintegrated into the display or the mobile device in FIG. 1 to providespecial illumination to the user's finger for optical sensing. Theoptical wavelength of such light sources for optical sensing can beselected to meet the optical sensing requirements. For example, one ormore red light sources may be used to illuminate red light onto theuser's finger for optical sensing of blood information of the user. Theillumination light from such other light sources can be modulated toimprove the optical sensing detection. For example, the display lightsources can be modulated with a proper pattern so as to rejectbackground light during detection. The sensor module can detectfingerprint, heartbeat, and blood flow speed etc. If specifiedwavelengths light sources are integrated into the display, the sensormodule can monitor other bio-parameters, such as glucose and degree ofblood oxygen saturation.

FIGS. 2A and 2B are block diagrams of an example of a fingerprintsensing module in a visible fingerprint sensor package for installing ina mobile device, such as the mobile device of FIG. 1. FIG. 2A is atop-down view of the fingerprint sensing module in the visible package.FIG. 2B is a side view along a direction that is parallel to the displayscreen surface showing an exemplary fingerprint sensing operation basedon detecting returned light by the optical detector array 23. As shownin FIG. 2A, the display assembly 10 with the integrated finger propertysensor includes a specified display zone 29 for fingerprint detection.Reference number 62 represents a finger pressing on the sensor. Similarto FIG. 1, the detector array 23 disposed near the specified displayzone 29 includes multiple detector elements including detector elementsin the fingerprint detector zone 33 for fingerprint sensing andfingerprint property detection responsive to the finger 62 pressing onthe sensor. The detector array 23 also includes detector elements in theenvironment and blood flow zones 41 and 43 for environment and bloodflow speed detection. Examples of the detector elements of the detectorarray 23 include optical devices such as photodiodes.

As shown in FIG. 2B, the display assembly 10 with an integrated fingerproperty sensor component includes a cover glass 50 and other displaylayers 54. Display elements 71 and 73 of the display screen within thedisplay assembly 10 can be disposed in the display layers 54 and areoperated to emit modulated light to display images on the display. Inaddition to their functions for displaying images, the light from thedisplay elements 71 and 73 exiting the top surface of the display screenin the specified display zone 29 for fingerprint detection near the edgeof the display screen is also used to illuminate a user's finger pressedin the zone 29 to create reflected or scattered light from the fingertowards the optical detector array 23 for fingerprint sending and otheroptical sensing operations. Examples of display elements 71 and 73include various types of light emitting devices such as light emittingdiodes (LEDs) and organic LEDs (OLEDs). Also, the detector array 23 isdisposed below the cover glass 50. Light beams 80 and 82 emitted fromthe display elements 71 and 73 interface with the cover glass 50, someof which pass through the top glass 10 and interface with differentparts of the finger 60. For example, at least some of the light beam 80emitted from display element 71 can pass through the cover glass 50 andinterface with finger skin ridge 61. A portion of the light beam 80 thatinterface with the finger skin ridge 61 is light 83 that is coupled orabsorbed into the finger tissues 60. Another portion of the light beam80 is a reflected light 81 that reflects off of the cover glass 50.

A portion of the light beam 82 emitted from the display element 73passes through the cover glass 50 and interfaces with finger skin valley63. The portion of the light beam 82 that interfaces with the fingerskin valley 63 is shown as light 89 that is coupled or absorbed intofinger tissues 60. Another portion of the light beam 82 is shown as areflected light 85 that reflects off of the cover glass 50. Yet anotherportion of the light beam 82 is shown as a finger skin reflected light87 that reflects off of the finger skin valley 63. Yet another portionof the light beam 82 ends up as scattered light 91 that scatters intothe detector array 23, such as photodiodes.

The display elements 71 and 73 in the specified zone 29 and the detectorelements in zone 33 are used to measure the fingerprint. The detectorelements in zone 41 or 43, or both 41 and 43 are used to monitor theenvironment light illumination. The display elements in the specifiedzone 29 are formed in one or more patterns appropriate for fingerprint,environment, and blood flow detection. For example, display elements inzone 29 can be divided into small groups, each group having anappropriate number of detector elements. The small groups of detectorelements can be turned on in turn to illuminate the finger placed on orclose to the sensor zones. The detector elements in the detector array23 detect the scattered light 91 scattered from the finger. The detectorsignals from the detector array 23 elements in zone 33 carry thefingerprint information. The detector signals from the detector elementsin zone 41, 43, or both are used to calibrate the fingerprint signalfrom zone 33 so as to eliminate the influence of the environment lightincluding the influence from other display zones.

As illustrated in FIGS. 2A and 2B, responsive to a finger placed on thedisplay screen and the photodiode area, display elements or lightsources 71 and 73 can emit light toward the finger to performfingerprint and fingerprint properties detection. Finger skin'sequivalent index of refraction is about 1.43 at 633 nm. Typical barecover glass index of refraction is about 1.51. When some of the displayelements 71 are illuminated at the finger skin ridge locations 61, thefinger ridge-cover glass interface reflected light 81 consumes verylittle power (˜0.1%) of the incident light 80. The majority of the lightbeam 80 is transmitted 83 into the finger tissues 60. A portion of thelight 80 is scattered 91 into the photo diode array 23.

When some of the display elements 73 are illuminated at the finger skinvalley locations 63, the cover glass surface reflects about 3.5% of theincident light 82 as reflected light 85, and the finger valley surfacereflects about 3.3% of the incident light as reflected light beam 87. Intotal, about 6.8% of the light 82 is lost by the surface reflection. Themajority of the light 82 is transmitted 89 into the finger tissues 60. Aportion of the light 82 is scattered 91 into the photo diode array 23.The surface reflection ratio difference between the finger valley andfinger ridge carries the fingerprint map information.

The display elements 71 and 73 can be turned on in sequence using amodulation pattern, for example, with different codes at differentlocations. Also, the detector arrays 23 of photodiodes can besynchronized with the display scanning. The modulation pattern of thedisplay elements 71 and 73, the detector array 23 synchronization, orboth can be used to acquire a sequence of signals. The sequence ofsignals can be demodulated to acquire a map of the finger ridges andvalleys by comparing the amplitudes of the signals.

When the distance between the illuminated display elements and thedetectors (e.g., photodiodes) changes, the light absorption of thefinger tissues also changes so that the photodiode detected light poweris affected. Adjusting the brightness of the display elements 71 and 73can calibrate or eliminate the influence of the change in the distancebetween the illuminated display elements and the detectors. For example,the display elements that are further away from the detector array canbe illuminated to be brighter than display elements that are closer tothe detector array 23. Due to the divergence of the display elementemitted light beam, and to enhance the fingerprint image contrast, thedisplay elements, such as RGB (red green blue) elements are set close tothe finger skin or the light beam is collimated.

FIG. 3 is a block diagram of an exemplary fingerprint sensing module ina visible package for heartbeat sensing. The fingerprint sensing moduleincludes a display assembly 10 with in integrated finger property sensorcomponent. Detector array 23 is disposed near the display assembly andpositioned to receive reflected or scatter light caused by a contact tothe top cover surface by a finger 62. A specified display zone 29 forfingerprint detection is designated on the display assembly 10 at alocation within the display screen near the detector array 23. Inresponse to a touch, the display pixels in the display zone 29 areturned on to project light towards the top cover surface of the displayto illuminate the top cover surface area within the display zone 29 toilluminate the user's finger 62. A portion of the reflected or scatteredlight from the user's finger 62 in touch with the top cover surface inthe display zone 29 can reach the nearby detector array 23. The detectorarray 23 includes detector elements in the fingerprint detector zone 33for fingerprint and fingerprint property detection. Detector elements inzones 41 and 43 are for environment and blood flow speed detection.

To detect the heartbeat signals, more display elements are turned onsimultaneously so as to generate sufficient incident light power for theoptical sensing of the heartbeat signals. The finger tissue lightabsorption ratio varies with the blood flow which is controlled by theheartbeat. The light absorption fluctuation signal reflects theheartbeat rate. In the disclosed technology, the light wavelength of thelight emitted by the display elements may be selected to optimize thedetection. Also, the light illumination may be modulated at a frequencyso as to further reduce the influence of the environment. For example,the specified display zone can be operated at a very high frame rate soas to realize the modulation. In some implementations, the light sourcesof the display can be modulated. In some implementations, extramodulated light sources can be integrated into the display assembly 10.Performing heartbeat sensing simultaneously with the fingerprintacquiring can greatly improve the secure access of the mobile device bydifferentiating between a fake fingerprint and a fingerprint from a livefinger.

FIG. 4 shows an exemplary fingerprint sensing module in a visiblepackage for blood flow speed sensing. The fingerprint sensing moduleincludes a display assembly 10 with in integrated finger property sensorcomponent. Detector array 23 is disposed near the display assembly andpositioned to receive a finger 101 to be monitored. The detector array23 includes detector elements in the fingerprint detector zone 33 forfingerprint and fingerprint property detection. Detector elements inzones 41 and 43 are for environment and blood flow speed detection. Thedisplay assembly 10 also includes specified zones 37 and 39 within thedisplay screen area for blood flow speed detection.

In FIG. 4, the fingerprint sensor assembly is structured to receive thefinger on the sensor area 101, with one side overlapping with displayelements in zone 37 and 39 and the other side overlapping with thedetectors (e.g., photodiodes) in zone 41 and 43 respectively. Displayelements in zone 37 and detectors in zone 41 cooperate to measure thepulse signals at one position of the finger. Display elements in zone 39and detectors in zone 43 cooperate to measure the pulse signals at theother position of the finger. The pulsation signal is generated in thearteries. In FIG. 4, a finger can be placed along the direction of thetwo zones 37 and 39 as illustrated. The blood flowing in the arteries isdetected at the left zone 37 and the right zone 39 representing theblood flowing across the zones from the left zone 37 to right zone 39.When the blood flows back from the right zone 39 to the left zone 37,the blood flows in the veins and no pulsation signal is obvious. Bycomparing the pulsation delay time between the two locations 37 and 39,the blood flow speed information can be acquired. Because blood pressureis correlated with blood flow speed, the sensor shown in FIG. 4 can alsomonitor the blood pressure.

In some implementations, the illumination light emitted by the displayelements may be modulated to reduce or eliminate background noise. Forexample, the emitted light can be modulated at a predetermined frequencyso as to further reduce the influence of the environmental conditions.Frequency modulating the emitted light can include operating the displayelements in the specified display zone at very high frame rate toachieve a desired modulation. Also, the display light sources can bemodulated. In some implementations, specified modulated light sourcescan be integrated into the display assembly 10.

FIG. 5 is a graph showing an exemplary signals from blood flow speedsensing performed in FIG. 4. FIG. 5 shows that there is a time delaybetween two positions (e.g., Position 1 and Position 2) along a fingerin the signals measured by the sensor module in FIG. 4. The typicalblood flow speed in a person is approximately 20 cm per second in thefinger's major arteries. If the distance between the two positions isapproximately 20 mm, the time delay is approximately 100 ms. This timedelay can be detected using the fingerprint sensor module disclosed inFIG. 4 and other designs in this document.

Invisible Optical Fingerprint Sensor Package

The above disclosed optical sensing functions associated withfingerprint sensing and other biometric marker sensing can bealternatively achieved by using an “invisible” optical fingerprintsensor package to match one or more detector arrays (e.g., an arraycontaining photodiodes) directly under the display so that the opticalfingerprint sensor package is underneath the device display screen.Because the optical sensing module are now underneath of the displayscreen and is hidden from the plain view from the top surface of themobile phone, this design is an “invisible” optical fingerprint sensorpackage. Implementations of this under-screen optical fingerprintsensing package may be used to eliminate the need for a window openingor designated area for the fingerprint sensor module so that the entiretop surface of the mobile device may be used for enlarging the displayscreen size while still providing optical sensing of fingerprints andother biometric marker measurements.

FIG. 6 is a block diagram showing an example of an invisible opticalfingerprint sensor package in a mobile device. FIG. 6 includes threedifferent views in FIGS. 6A (top view from the display side of themobile device), 6B (side view along the line B-B′ in FIG. 6A to showdifferent layers in the fingerprint sensor) and 6C (showing some detailsof example of optical detector arrangement of the fingerprint sensor).Referring to FIG. 1A, the mobile device utilizing the invisible packageof the fingerprint sensor includes a display assembly 10 with anintegrated finger property sensor component. Reference number 12indicates one or more other sensors on the mobile device. The mobiledevice can also include user input mechanisms such as side buttons 14,16 prepared for the smart terminals. Reference number 21 represents thefingerprint sensor zone, where the fingerprint sensor is located belowthe display screen. Located near the fingerprint sensor zone 21 is aspecified display zone 29 within the display screen for fingerprintdetection for receiving and illuminating a user's finger and thus may bereferred to as the illumination zone 29. The zone 21 may overlap orpartially overlap the illumination zone 29 to receive part of reflectedor scattered light from the illuminated finger in touch with theillumination zone 29. As shown in FIG. 6B, reference number 123represents an optical detector array associated with the fingerprintsensor. Disposed above the detector array 123 are receiving optics 124,which are the optical detectors for detecting light reflected off of atarget, such as a finger. A back board with integrated circuitry isdisposed below the detector array. In some implementations, the detectorarray 123 can be integrated into the backboard 125. The backboard 125can be disposed over a flexible printed circuit (FPC) 127.

The detector array 123 includes multiple detector elements arranged indifferent detector zones including fingerprint detector zone 133 forfingerprint and fingerprint property detection and environment and bloodflow detector zones 141 and 143 for environment and blood flow speeddetection. Referring to FIG. 6C, reference numbers 37 and 39 representspecified zones over an enhanced cover glass 50 for blood flow speeddetection. The display assembly 10 includes the enhanced cover glass 10and other display layers 54 disposed below the enhanced cover glass 50.A support glass 56 may be disposed under the cover glass 50 in someimplementations.

As shown in FIG. 6B, the fingerprint sensor zone 21 and the associatedoptical detector sensor module 123 for detecting fingerprint sensing andother optical signals are located under the top cover glass or layer 50so that the top surface of the cover glass or layer 50 serves as the topsurface of the mobile device as a contiguous and uniform glass surfaceacross both the display screen of the display assembly 10 (containingthe specified display zone 29 for fingerprint detection near the edge ofthe display screen) and the fingerprint sensor zone 21 and theassociated optical detector sensor module 123.

The invisible optical fingerprint sensor package can be used in anydevices with display or similar light sources. The display elements ofseveral zones 29, 37, 39 are used to illuminate the user's finger.Several integrated detector (e.g., photodiode) arrays 123, 133, 141, 143underneath the display screen are used to detect the light scatteredfrom the finger and detect the light in the environment. The photodiodearrays 123, 133, 141, 143 are packaged close to the specified displayzones 29, 37, 39 and fixed under the display assembly 10 so as toachieve a high light detection efficiency. The positions of thespecified display zones 29, 37, 39 are not limited to the examples shownin FIG. 6, and can be adjusted to be placed at various suitablelocations to perform fingerprint detection and other optical sensingoperations.

Like the design in FIG. 1, the illumination light for the opticalfingerprint sensing and other optical sensing can be provided indifferent ways. In some implementations, the display pixels may be usedto provide the illumination light to the finger. In otherimplementations, additional light sources can be integrated into thedisplay to provide special illumination. Also, the added light sourcescan be modulated. For example, the display light sources can bemodulated using an appropriate pattern so as to reject background lightduring the detection.

The sensor module as shown in FIG. 6 can detect fingerprints andadditional biometric markers, such as heartbeat and blood flow speed. Iflight sources of certain specified wavelengths are integrated into thedisplay, the sensor module can further monitor additionalbio-parameters, such as glucose and degree of blood oxygen saturation.

FIGS. 7A and 7B are block diagrams of an exemplary fingerprint sensingmodule in an invisible package for installing in a mobile device, suchas the mobile device of FIG. 6. FIG. 7A is a top-down view of thefingerprint sensing module in the visible package. FIG. 7B is a blockdiagram showing a cross sectional view of the invisible opticalfingerprint sensor package along the line B-B′ in FIG. 7A and furtherillustrates an exemplary fingerprint sensing operation. As shown in FIG.7A, the display assembly 10 with the integrated finger property sensorincludes a specified display zone 29 for fingerprint detection.Reference number 62 represents a finger pressing on the sensor. Similarto FIG. 6, the detector array 123 disposed near the specified displayzone 29 includes multiple detector elements including detector elementsin the fingerprint detector zone 133 for fingerprint and fingerprintproperty detection responsive to the finger 62 pressing on the sensor.The detector array 123 also includes detector elements in theenvironment and blood flow zones 141 and 143 for environment and bloodflow speed detection. Examples of the detector elements of the detectorarray 123 include optical devices such as photodiodes.

As shown in FIG. 7B, the display assembly 10 with an integrated fingerproperty sensor component includes a cover glass 50 and other displaylayers 54. Display elements 71 and 73 can be disposed in the otherdisplay layers 54. Examples of display elements 71 and 73 includevarious types of light emitting devices such as light emitting diodes(LEDs). Also, the detector array 123 is disposed below the cover glass50. Light beams 80 and 82 emitted from the display elements 71 and 73interface with the cover glass 50, some of which pass through the topglass 10 and interface with different parts of the finger 60. Forexample, at least some of the light beam 80 emitted from display element71 can pass through the cover glass 50 and interface with finger skinridge 61. A portion of the light beam 80 that interface with the fingerskin ridge 61 is light 183 that is coupled or absorbed into the fingertissues 60. Another portion of the light beam 80 is a reflected light181 that reflects off of the cover glass 50.

A portion of the light beam 82 emitted from the display element 73passes through the cover glass 50 and interfaces with finger skin valley63. The portion of the light beam 82 that interfaces with the fingerskin valley 63 is shown as light 189 that is coupled or absorbed intofinger tissues 60. Another portion of the light beam 82 is shown as areflected light 185 that reflects off of the cover glass 50. Yet anotherportion of the light beam 82 is shown as a finger skin reflected light187 that reflects off of the finger skin valley 63. Yet another portionof the light beam 82 ends up as scattered light 191 that scatters intothe detector array 123, such as photodiodes.

The display elements 71 and 73 in the specified zone 29 and the detectorelements in zone 133 are used to measure the fingerprint. The detectorelements in zone 141, 143, or both 141 and 143 are used to monitor theenvironment light illumination. The display elements in the specifiedzone 29 are formed in one or more patterns appropriate for fingerprint,environment, and blood flow detection. For example, display elements inzone 29 can be divided into small groups, each group having anappropriate number of detector elements. The small groups of detectorelements can be turned on in turn to illuminate the finger placed on orclose to the sensor zones. The detector elements in the detector array23 detect the scattered light 91 scattered from the finger. The detectorsignals from the detector array 23 elements in zone 133 carry thefingerprint information. The detector signals from the detector elementsin zone 141, 143, or both are used to calibrate the fingerprint signalfrom zone 133 so as to eliminate the influence of the environment lightincluding the influence from other display zones.

As illustrated in FIGS. 7A and 7B, responsive to a finger placed on thedisplay screen and the photodiode area, display elements or lightsources 71 and 73 can emit light toward the finger to performfingerprint and fingerprint properties detection. Finger skin'sequivalent index of refraction is about 1.43 at 633 nm. Typical barecover glass index of refraction is about 1.51. When some of the displayelements 71 are illuminated at the finger skin ridge locations 61, thefinger ridge-cover glass interface reflected light 181 consumes verylittle power (˜0.1%) of the incident light 80. The majority of the lightbeam 80 is transmitted 83 into the finger tissues 60. A portion of thelight 80 is scattered 191 into the photo diode array 123.

When some of the display elements 73 are illuminated at the finger skinvalley locations 63, the cover glass surface reflects about 3.5% of theincident light 82 as reflected light 185, and the finger valley surfacereflects about 3.3% of the incident light as reflected light beam 187.In total, about 6.8% of the light 82 is lost by the surface reflection.The majority of the light 82 is transmitted 189 into the finger tissues60. A portion of the light 82 is scattered 191 into the photo diodearray 23. The surface reflection ratio difference between the fingervalley and finger ridge carries the fingerprint map information.

The display elements 71 and 73 can be turned on in sequence using amodulation pattern, for example, with different code at differentlocations. Also, the detector arrays 123 of photodiodes can besynchronized with the display scanning. The modulation pattern of thedisplay elements 71 and 73, the detector array 123 synchronization, orboth can be used to acquire a sequence of signals. The sequence ofsignals can be demodulated to acquire a map of the finger ridges andvalleys by comparing the amplitudes of the signals.

When the distance between the illuminated display elements and thedetectors (e.g., photodiodes) changes, the light absorption of thefinger tissues also changes so that the photodiode detected light poweris affected. Adjusting the brightness of the display elements 71 and 73can calibrate or eliminate the influence of the change in the distancebetween the illuminated display elements and the detectors. For example,the display elements that are further away from the detector array canbe illuminated to be brighter than display elements that are closer tothe detector array 123. Due to the divergence of the display elementemitted light beam, and to enhance the fingerprint image contrast, thedisplay elements, such as RGB (red green blue) elements are set close tothe finger skin or the light beam is collimated.

FIG. 8 is a block diagram of an exemplary fingerprint sensing module inan invisible package for heartbeat sensing. The fingerprint sensingmodule includes a display assembly 10 with in integrated finger propertysensor component. Detector array 123 is disposed near the displayassembly and positioned to receive a contact by a finger 62. A specifieddisplay zone 29 for fingerprint detection is disposed on the displayassembly 10 to be arranged near the detector array 123. The detectorarray 123 includes detector elements in the fingerprint detector zone133 for fingerprint and fingerprint property detection. Detectorelements in zones 141 and 143 are for environment and blood flow speeddetection.

To detect the heartbeat signals, more display elements are turned onsimultaneously so as to generate enough incident light power. The fingertissue light absorption ratio varies with the blood flow which iscontrolled by the heartbeat. The light absorption fluctuation signalreflects the heartbeat rate. In the disclosed technology, the lightwavelength of the light emitted by the display elements may be selectedto optimize the detection. Also, the light illumination may be modulatedat a frequency so as to further reduce the influence of the environment.For example, the specified display zone can be operated at a very highframe rate so as to realize the modulation. In some implementations, thelight sources of the display can be modulated. In some implementations,extra modulated light sources can be integrated into the displayassembly 10. Performing heartbeat sensing simultaneously with thefingerprint acquiring can greatly improve the secure access of themobile device by differentiating between a fake fingerprint and afingerprint from a live finger.

FIG. 9 is a block diagram of an exemplary fingerprint sensing module inan invisible package for blood flow speed sensing. The fingerprintsensing module includes a display assembly 10 with in integrated fingerproperty sensor component. Detector array 123 is disposed near thedisplay assembly and positioned to receive a finger 101 to be monitored.The detector array 123 includes detector elements in the fingerprintdetector zone 133 for fingerprint and fingerprint property detection.Detector elements in zones 141 and 143 are for environment and bloodflow speed detection. The display assembly 10 also includes specifiedzones 137 and 139 for blood flow speed detection.

As illustrated in FIG. 9, the fingerprint sensor assembly is structuredto receive the finger on the sensor area 101, with one side overlappingwith display elements in zone 137 and 139 and the other side overlappingwith the detectors (e.g., photodiodes) in zone 41 and 43 respectively.Display elements in zone 137 and detectors in zone 141 cooperate tomeasure the pulse signals at one position of the finger. Displayelements in zone 139 and detectors in zone 143 cooperate to measure thepulse signals at the other position of the finger. The pulsation signalis generated in the arteries. In FIG. 9, the blood flowing in thearteries is detected at the left zone 137 and the right zone 139representing the blood flowing across the zones from the left zone 137to right zone 139. When the blood flows back from the right zone 139 tothe left zone 137, the blood flows in the veins and no pulsation signalis obvious. By comparing the pulsation delay time between the twolocations 137 and 139, the blood flow speed information can be acquired.Because blood pressure is correlated with blood flow speed, the sensorshown in FIG. 9 can also monitor the blood pressure.

In some implementations, the illumination light emitted by the displayelements may be modulated to reduce or eliminate background noise. Forexample, the emitted light can be modulated at a predetermined frequencyso as to further reduce the influence of the environmental conditions.Frequency modulating the emitted light can include operating the displayelements in the specified display zone at very high frame rate toachieve a desired modulation. Also, the display light sources can bemodulated. In some implementations, specified modulated light sourcescan be integrated into the display assembly 10.

Total Optical Reflection Fingerprint Sensing

In another aspect, the total optical reflection effect at the coverglass of the mobile device can be used to acquire the fingerprintsignals. The detector array can be integrated directly under the displayin some situations such as OLED display, or fixed at the display edgepositions.

FIGS. 10A and 10B are cross-sectional and top-down views of an exemplaryfingerprint sensor module implementing a total reflection fingerprintsensing technique in an invisible optical fingerprint sensor package. Asshown in FIG. 10A, the display assembly 10 with the integrated fingerproperty sensor includes a specified display zone 29 for fingerprintdetection. Reference number 62 represents a location on the displaywhere the finger 60 can press on the sensor. Similar to FIGS. 1 and 6,the detector array 123 disposed near the specified display zone 29includes multiple detector elements including detector elements in thefingerprint detector zone 133 for fingerprint and fingerprint propertydetection responsive to the finger 60 pressing on the sensor at location62. The detector array 123 also includes detector elements in theenvironment and blood flow zones 141 and 413 for environment and bloodflow speed detection. Examples of the detector elements of the detectorarray 123 include optical devices such as photodiodes.

As shown in FIG. 10B, the display assembly 10 with an integrated fingerproperty sensor component includes a cover glass 50 and other displaylayers 54. Display element 73 can be disposed in the other displaylayers 54. Examples of display element 73 include various types of lightemitting devices such as light emitting diodes (LEDs). Also, thedetector array 123 is disposed below the cover glass 50 and underneaththe display screen layers 54.

As illustrated in FIGS. 10A and 10B, a finger 60 is pressing on thedisplay screen at the photodiode area 62. The finger skin's equivalentindex of refraction is about 1.43 at 633 nm. The typical bare coverglass index of refraction is about 1.51. If the cover glass 50 and thedisplay layers 54 are glued together without an air gap, the lightemitted by the display element 73 with a large incident angle will betotally reflected at the screen-air interface.

When the display element 73 is turned on, the divergent light beams canbe divided into two groups: the central beam 82 that is not totallyreflected, and outer beams 201, 202, 211, 212 that are totally reflectedwhen nothing touches the screen surface. For the central light beams 82,the screen surface reflects about 0.1%˜3.5% of beam 82 as light beam 185that is received by photo diodes 193, the finger skin reflects about0.1%˜3.3% of beam 82 as light beam 187 that may be received by somephoto diodes 195. The reflection difference is dependent at least onwhether the light beams 82 meets with finger skin ridge 61 or valley 63.The rest of the light beam 82 is coupled 189 into the finger tissues.

For outer light beams 201 and 202, the screen surface reflects ˜100% aslight beams 205 and 206 respectively if nothing touches the screensurface. When the finger skin ridges touch the screen surface near lightbeams 201 and 202, most of the light power is coupled into the fingertissues 60 as light beams 203 and 204.

For outer light beams 211 and 212, the screen surface reflects at a highreflectivity (e.g., near 100%) as light beams 213 and 214 respectivelyif nothing touches the screen surface. When the finger touches thescreen surface and the finger skin valleys happen to be at light beams211 and 212 positions, no light power is coupled into finger tissues 60.

All the light beams that are coupled into finger tissues 60 may berandomly scattered to form a low-contrast light 191 and received bymultiple photo diodes 207, 215 etc. In the outer light beam illuminatedarea, the finger skin ridges and valleys cause obvious reflectiondifference that is detected in the corresponding receiving photo diodes.Because the coordinates of the display emitting elements and theprocessing photo diodes are known, the fingerprint signals are acquiredby comparing the differences based on the known locations of the displayelements and detector elements.

The detector array can be disposed under the display, or disposedbesides the display, or any locations where the total reflection can bedetected. The detector array can be glued onto the display layers, orattached to the display layer with the help of optical wedge, prism,lens etc.

FIGS. 11A and 11B are cross-sectional and top-down views of an exemplaryfingerprint sensor module implementing total reflection fingerprintdetection scanning in an invisible optical fingerprint sensor package.As shown in FIGS. 11A and 11B, the cover glass 50 and the other displaylayers 54 are glued together to enable total reflection at the coverglass surface. When the display element (e.g., light emitting element)241 and the detector element (e.g., processing photodiodes) 233 aredefined with respect to the distance between the cover glass 50 surfaceand the respective display elements and detector elements (e.g. definedas H1, H2, and L respectively), the coordinate LF of the correspondingtotal reflection zone 231 can be calculated. When the size A of theemitting element 241 and the size D of the processing photo diodes 233are known, the size W of the corresponding total reflection zone 231 canalso be calculated. In various implementations, W can be smaller than Dif the emitting elements size is small enough. For an array ofillumination pixels 241 in the display screen structure 54, there is acorresponding area 245 which is sufficiently away from the illuminationpixels 241 so that the light from the illumination pixels 241 is totallyreflected. The optical detectors for fingerprint sensing and otheroptical sensing can be placed under the display screen structure 54 inthe locations within the paths of the totally reflected light beams fromthe area 245.

Expanding the size A of the emitting element 241 and the size D of theprocessing photodiodes 233, the size W of the corresponding totalreflection zone 231 is also expanded. This implies that the resolutionis reduced. Assuming the environment is air with a refraction index ofnearly 1, and the cover glass' refraction index is n, the minimum totalreflection incident angle θ can be calculated:

$\theta = {\sin^{-}1{\left( \frac{1}{n} \right).}}$

As a result, the central light beams zone 243 can be calculated. Therest of the positions are located in the total reflection zone 245. Forexample, if n=1.51, and H1=0.6 mm, for a point light source, thediameter of the central light beams zone is about 1.06 mm. If H2 isgiven, the closest photo diode distance Lmin can be calculated. Forexample, when H2=1.2 mm, the minimum Lmin is about 1.59 mm.

When one display element is turned on, multiple point fingerprintsignals can be simultaneously acquired with multiple element detectorarray 123. Or, for a known photo diode, multiple point fingerprintsignals can be simultaneously acquired by sequentially lighting onmultiple display elements.

FIGS. 12A, 12B, and 12C represent an exemplary fingerprint sensor moduleinstalled in a mobile device and implementing total reflectionfingerprint detection scanning. As shown, cover glass 50 and otherdisplay layers 54 are glued or otherwise engaged together so that totalreflection can happen at the cover glass surface. Underneath of thedisplay screen structure 54 is the 2D detector array 261 with multiplephotodetectors such as a group of photodetectors 233 and another groupof photodetectors 235. One or more light emitting elements 241 areturned on when the device is touched. Each light beam from the lightemitting element 241 is totally reflected in a zone 263 within the totalreflection zone 245 to a corresponding direction towards the detectorarray 261 placed underneath the display screen structure 54. Forexample, light beams 221 and 223 are reflected to light beams 222 and224 respectively. The reflected light beams 222 and 224 are received byphoto diodes 233. Similarly, other light beams from the light emittingelement (i.e., light source) 241 are received by photo diodes 235.

The 2D detector array 261 is used to detect the total reflected lightfrom the zone 263 or other zones in the zone 245. The fingerprint in thecorresponding detection zone 263 can be detected. Because of the lightbending by the reflection and because of the divergence of the probinglight beams, there is some distortion in the fingerprint image. Thisdistortion can be corrected based on the coordinates of the light sourceand the photo diodes. Symmetrically, by using single small-sized photodiodes, and scanning the light sources in 2 dimensions, the fingerprintcan also be detected.

In some implementations, the light sources or additional light sourcescan be packaged at other positions, such as 242 at one end of thedisplay. The added light sources can be partial of the display, ordiscrete light sources such as LEDs. When the light sources are far awayfrom the detector array, the distortion is reduced.

As described above, the 2D detector array position is also flexible. Anylocations where the total reflection can be received may be used toplace the detector array.

In some applications, such as in smartphone, tablet etc., thefingerprint distortion can be eliminated with the help of the touchsensor that identifies the touch finger locations.

FIG. 13 shows an example of an image representing a result of a totalreflection fingerprint detection scanning in operating the device shownin FIGS. 11A, 11B, 12A, 12B and 12C. This resultant image of thefingerprint obtained using the total reflection is captured by a 2Doptical detector array for the subsequent fingerprint sensing operation.The resultant fingerprint image reveals an enhanced quality of thefingerprint image.

FIGS. 14A, 14B, and 14C represent another exemplary fingerprint sensormodule installed in a mobile device and implementing total reflectionfingerprint detection. As shown in FIGS. 14A, 14B, and 14C, cover glass50 and other display layers 54 are glued together to enable totalreflection at the cover glass surface. The light emitting element (i.e.,light source) 241 is turned on. Each light beam from the source 241 isreflected to a proper direction. For example, light beams 221 and 223are reflected to light beams 222 and 224 respectively. The reflectedlight beams 222 and 224 are received by photo diodes 233. Similarly,other light beams from the light source 241 are received by the photodiodes 235.

When a linear detector array 271 is used to detect the total reflectedlight, the fingerprint in the corresponding detection line 273 can bedetected. Because of the light bending by the reflection and because ofthe divergence of the probing light beams, there is some distortion inthe fingerprint image. This distortion can be corrected based on thecoordinates of the light source and the photo diodes. When the lightsource 241 is scanned along linear directions 275, the detection line273 is also scanned to cover a 2D detection zone.

In some implementations, the light sources or additional light sourcescan be packaged at other positions, such as 281 at one end of thedisplay. These light sources can be partial of the display, or discretelight sources such as linear LED array. When the light sources are faraway from the detector array, the distortion is reduced.

The emitted light from the light source elements can be modulated basedon a code system, such as any one of the spread spectrum code systemsincluding Code Division Multiple Access (CDMA) so as to simplify thefingerprint image detection. Any of the CDMA and hybrid CDMA techniquescan be used. For example, Direct Sequence Spread Spectrum (DSSS) andFrequency Hopping Spread Spectrum (FHSS) including adaptive FHSS can beused. Using any of the spread spectrum techniques, the light emittedfrom the light source elements can be modulated using distinctorthogonal pseudorandom spreading codes, different frequencies,amplitudes, phases, or any combination of them.

In addition, the detector array position is flexible. Any locationswhere the total reflection can be received may be used to place thedetector array.

In some applications, such as in smartphone, tablet etc., thefingerprint distortion can be eliminated with the help of the touchingsensor that can identify the touch finger locations.

FIGS. 15A and 15B represent another exemplary fingerprint sensor moduleinstalled in a mobile device and implementing total reflectionfingerprint sensing. The underlying optical detector module is partiallyunder the display screen structure 54 as being “invisible” while a partof the optical detector module is outside the footprint of the displayscreen structure 54 as being “visible” so that this package is apartially visible and partially invisible package. The finger printsensor of FIGS. 15A and 15B includes detector array(s) 123 disposed nearthe fingerprint sensing zone 21 and a specified display zone 29 forfingerprint detection. A cover glass 50 and other display layers 54 areglued together to enable total reflection at the cover glass surface.Display elements 72, 74 are embedded in the other display layers 54 andcan emit light beams 221, 223 used for fingerprint detection. Theemitted light beams 221 and 223 are nearly 100% reflected as cover glasstotal reflected lights 222 and 224, which are captured or detected byphotodiodes 233 and 235. A connection block 251 is glued to the displaylayers so that the dimension H2 (distance between the cover glass 50surface and the display elements) is increased. As a result, the minimumL (distance between the light source and the detector) is alsoincreased. Therefore, the processing photo diodes can be shifted to outof the display zone. Because the photo diodes only detect the totalreflection light, no window is required on the cover glass color layers.Thus, this design is also considered as an invisible package.

In some implementations, the connection block 251 can be of a wedgeshape or be a prism. With the help of the connection block, the detectorarray may be tilted at a proper angle so as to reduce the fingerprintimage distortion.

FIG. 16 is a process flow diagram showing an exemplary process 1600 forfingerprint detection. Sensor detection can be initiated (1602) toactivate the related function modules, including the light sources,detector array, modulators, processing circuit, memory, touching sensorsetc. A modulating or encoding operation (1604) can be performed tocontrol the light sources to emit light beams that carry the modulationsignal information including amplitude, phase shift, frequency change,or a combination. A signal acquisition (1606) can be performed by havingthe detector array receive the optical signals. Demodulation (1608) isperformed to have the processing circuits amplify the signals and checkthe electric effects. Detection result evaluation (1610) is performed toachieve the fingerprint image, signal strength time dependence(heartbeat, blood flow speed), and compare the signals with thecriterion.

To determine whether the detection should be repeated (1612), thefollowing operations are performed. Once the initial detection result isevaluated, the processor makes decisions on following tasks: Forexample, the processor sends the fingerprint image to memory for switchcontrol or security control, to determine the user blood flow speed etc.According to the application of the detection results, the processordetermines whether to continue the detection. When the processordetermines that the detection will not be repeated or continued, a newjob activation (1616) can be performed according to the detectionresult, to activate various operations.

Total Reflection Touch Sensing-Refractive Index Matching

Cover glass total reflection effect can be used to acquire thefingerprint and touch signals. The detector array can be integrateddirectly under the display in some situations such as OLED display, orfixed at the display edge positions. In case of the situation that thelight sources are too far away so that the local incident angle is toobig, the refractive index matching can guarantee the performance of thisconcept.

FIG. 17 shows a cross section of another device design where an opticalsensing module is provided total reflection touch sensing-refractiveindex matching technique FIGS. 18A and 18B show an example of a deviceimplementing the design in FIG. 17 based on the total reflection touchsensing-refractive index matching technique. As shown in FIGS. 17, 18Aand 18B, cover glass 50 and other display layers 54 are glued togetherso that total reflection can happen at the cover glass surface. The glueto connect the cover glass 50 and the display layers 54 has refractiveindex n1, cover glass 50 has a refractive index of n, and the touchingmaterial 60 a (finger etc.) has a refractive index of n2. If the localincident angle θ is greater than the total reflection angle decided by nand n2, no light can be coupled out by the touching material. When n1 isnot greater than n2, any local incident angle θ is acceptable. Thus, aproper glue material can be used to match the refractive index helpsimproving the performance. Each light beam from the emitting elements241 is reflected to a proper direction. For example, light beams 301 and303 are reflected to photo diodes 235. If a linear detector array 271 isused to detect the total reflected light, the fingerprint in thecorresponding detection line 273 can be detected. When the light sources241 are scanned along directions 275, the detection line 273 is alsoscanned to cover a two dimensional (2D) detection zone. When thedetector array 271 is 2D, the corresponding detection zone 273 is also2D. When the light source 241 is scanned, the detection zone 273 is alsoscanned.

FIGS. 19A, 19B, and 19C are diagrams showing an example of a totalreflection touch sensing-refractive index matching technique. As shownin the FIGS. 19A, 19B, and 19C, cover glass 50 and other display layers54 are assembled together. A connection block 252 is glued to the cover50 under the sensing window 50 a so that total reflection can occur atthe sensing window. The connection block 252 has a refractive index ofn1, cover glass 50 has a refractive index of n, and the touchingmaterial 60 a (finger etc.) has a refractive index of n2. When n1 is notgreater than n2, any local incident angle is acceptable. Using a properconnection block material to match the refractive index can improve theperformance. Each light beam from the emitting elements 241 is reflectedto a proper direction. For example, light beams 305 and 307 arereflected to photo diodes 235. When a linear detector array 271 is usedto detect the total reflected light, the fingerprint in thecorresponding detection line 273 can be detected. When the light sources241 are scanned along directions 275, the detection line 273 is alsoscanned to cover a 2D detection zone. Also, when the detector array 271is 2D, the corresponding detection zone 273 is also 2D. When the lightsource 241 is scanned, the detection zone 273 is also scanned.

Fingerprint Sensor Technologies

In the above examples for detecting fingerprints and other biometricparameters, the fingerprint sensor is based on optical sensing. Opticalfingerprint sensing may be substituted in some implementations by otherfingerprint sensors such as capacitive fingerprint sensors or a hybridfingerprint sensor with both optical sensing and capacitive sensing forsensing fingerprints. Accordingly, fingerprint sensor modules asdisclosed in this patent document can be implemented using one or acombination of various sensing technologies including self-capacitivesensing, mutual capacitive sensing, and optical sensing among others.The disclosed technology for detecting a live finger is not dependent ona particular type of sensing technology, and one or a combination of thevarious sensing technologies can be incorporated.

For example, FIG. 20A is a block diagram of an exemplary fingerprintsensor device 2000 implementing self-capacitive sensing with activesensor pixel and amplification. The fingerprint sensor device 2000 isone example of a fingerprint sensor that implements self-capacitivesensing and can be used to replace an optical fingerprint sensor shownin FIGS. 1 through 19C. For example, the capacitive sensing fingerprintsensor in FIG. 20A may be used in combination with the optical sensors41 and 43 by placing it at one end of the display screen structure 54underneath a thin top cover glass layer 50 in the visible fingerprintsensor package as shown in FIG. 1. For another example, the capacitivesensing fingerprint sensor in FIG. 20A may be used in combination withthe optical sensors 41 and 43 by placing it in a separate sensorstructure as such as a button like fingerprint sensor structure in anopening of the top glass cover 50 in some mobile phone designs. Theself-capacitive fingerprint sensor device 2000 includes a sensor chip2002 disposed over a substrate carrier 2004 and a protective film orcover layer 2006 disposed over the sensor chip 2002. The protective filmor cover layer 2006 can include an insulator or dielectric material suchas glass, silicon dioxide (SiO₂), sapphire, plastic, polymer, othersubstantially similar materials. The protective film or cover layer 2006can be present to protect the sensor chip 2002 and to function as a partof a dielectric layer between a surface of a finger 2001 and conductivesensing electrodes of individual sensor pixels in the sensor chip 2002.The protective film or cover layer 2006 is an optional layer dependingon the application of the fingerprint sensor device 2000. In someimplementations, the fingerprint sensor device 2000 can be disposedthrough an opening of a top cover glass of an electronic device such asa mobile phone or under a top cover glass of the electronic device. Whenused in the under-the-glass application, the protective film or cover2006 is not needed because the top cover glass of the electronic devicewill function to protect the sensor chip 2002 and act as the dielectriclayer.

The sensor chip 2002 includes an array of sensor pixels that incombination senses or captures fingerprint data from the finger 2001 incontact with the protective film or cover layer 2006. An exemplarysensor pixel 2008 is shown in FIG. 20B. Each sensor pixel 2008 of thesensor chip 2002 generates an output signal (e.g., a voltage) based on acapacitance of a capacitor associated with a ridge or valley of thefinger 2001. The output signals when combined together can represent afingerprint image of the finger 2001. Higher the number of pixelsensors, greater the resolution of the fingerprint image.

FIG. 20C shows the circuit equivalent of the sensor pixel. Each sensorpixel in the array of sensor pixels of the sensor chip 2002 can besubstantially similar to the exemplary sensor pixel 2008. The exemplarysensor pixel 108 includes an operational amplifier 2022 to amplify acapacitance related signal (e.g., voltage signal) detected by theexemplary sensor pixel 2008. A sensor electrode 2012 that includes aconductive material, such as one of a variety of metals is electricallyconnected to a negative or inverting terminal of the amplifier 2022. Theamplifier configuration shown and described with respect to FIG. 20B(and other figures of this patent document) is just one example andother amplifier configurations are possible including a positivefeedback configuration. The sensor electrode 2012 and a local surface ofthe finger 2001 function as opposing plates of a capacitor CS 2030. Thecapacitance of the capacitor CS 2030 varies based on a distance betweenthe local surface of the finger 2001 and the sensor electrode 2012, thedistance between the two plates of the capacitor CS 2030. Thecapacitance is inversely proportional to the distance ‘d’ between thetwo plates of the capacitor CS 2030. The capacitance is larger when thesensor electrode 2012 is opposite a ridge of the finger 2001 than whenopposite a valley of the finger 2001.

In addition, various parasitic capacitors can be formed betweendifferent conductive elements in the exemplary sensor pixel 2008. Forexample, a parasitic capacitor CP1 2026 can form between the sensorelectrode 2012 and a device ground terminal 2014. Another parasiticcapacitor CP2 2017 can form between the local surface of the finger 2001and an earth ground 2016. Device ground is coupled to earth groundclosely. Yet another capacitor CF 128 can form between an outputconductor of the amplifier 2022 and the negative or inverting terminalof the amplifier 2022 and functions as a feedback capacitor to theamplifier 2022.

The positive terminal of the amplifier 2022 is electrically connected toan excitation signal Vin 2018. The excitation signal Vin 2018 can bedirectly provided to the positive terminal of a dedicated amplifier ineach sensor pixel. By providing the excitation signal Vin 2018 directlyto the positive terminal of the amplifier 2022, the exemplary sensorpixel 2008 becomes an active sensor pixel. In addition, providing theexcitation signal Vin 2018 directly to the positive terminal of theamplifier 2022 eliminates the need to include an excitation electrode,common to all sensor pixels, which reduces a conductive (e.g., metal)layer from the semiconductor structure of the sensor chip 2002. Inaddition, by providing the excitation signal Vin 2018 directly to theamplifier 2022, the excitation signal Vin 2018 is not applied directlyto the finger to avoid potentially irritating or injuring the finger2001. Moreover, because the excitation electrode for applying theexcitation signal directly to the finger is not used, all components ofthe fingerprint sensor device 2000 can be integrated into a singlepackaged device, and the entire fingerprint sensor device 2000 can bedisposed under the protective cover glass. With the entire fingerprintsensor device 2000 disposed under the protective cover glass, thefingerprint sensor device 2000 is protected from the finger and otherexternal elements that can potentially damage the fingerprint sensor.

The amplifier 2022 can generate an output signal based at least on thevariable capacitance of the variable capacitor CS 2030, and the outputsignal can contribute to the overall fingerprint data. The amplifier2022 can generate the output signal based at least on the variablecapacitance and feedback capacitance of the feedback capacitor CF withno additional non-parasitic capacitances contributing to the outputsignal. This is partly because, as described above, an additionalelectrode such as an external drive electrode is not uses in the sensorpixel 2008.

Fingerprint Sensor Technologies: Optical Sensors Integrated with Pixels

In another aspect of the disclosed technology, each sensing pixel of asensing pixel array of a fingerprint sensor device can be a hybridsensing pixel having a capacitive sensor for capturing fingerprintinformation and an optical sensor for capturing fingerprint informationincluding live finger detection as disclosed in this patent document.

FIGS. 21A and 21B show two examples of hybrid sensing pixel designs thatcombine capacitive sensing and optical sensing within the same sensingpixel.

FIG. 21A shows an example of a fingerprint sensor device 2100 thatincorporates a capacitive sensor in addition to an optical sensor foreach sensor pixel of an array of sensor pixels in capturing fingerprintinformation. By combining both capacitive sensors and optical sensors,fingerprint images obtained with the optical sensors can be used tobetter resolve the 3D fingerprint structure obtained with the capacitivesensors. For illustrative purposes, the structure shown in FIG. 21Arepresents one sensor pixel in an array of sensor pixels and each sensorpixel includes an optical sensor 2102 and a capacitive sensor 2114 thatare disposed next to each other within the same pixel.

The optical sensor 2102 includes a photodetector 2108 and a collimator2106 disposed over the photodetector 2108 to narrow or focus reflectedlight 2124 from finger 2102 toward the photodetector 2108. One or morelight sources, such as LEDs (not shown) can be disposed around thecollimator 2106 to emit light, which is reflected off the finger asreflected light 2124 and is directed or focused toward the correspondingphotodetector 2108 to capture a part of the fingerprint image of thefinger 2102. The collimator 2106 can be implemented using an opticalfiber bundle or one or more metal layer(s) with holes or openings. Thisuse of multiple optical collimators above the optical detector array maybe used as a lensless optical design for capturing the fingerprint imagewith a desired spatial resolution for reliable optical fingerprintssensing. FIG. 21A shows the collimator 2106 implemented using one ormore metal layers 2110 with holes or openings 2112. The collimator 2106in the layer between the top structure or layer 2104 and thephotodetectors 2108 in FIG. 21A includes multiple individual opticalcollimators formed by optical fibers or by holes or openings in one ormore layers (e.g., silicon or metal) and each of such individual opticalcollimators receives light ray 2124 in a direction along thelongitudinal direction of each optical collimator or within a smallangle range that can be captured by the top opening of each opening orhole and by the tubular structure as shown so that light rays incidentin large angles from the longitudinal direction of each opticalcollimator are rejected by each collimator from reaching the opticalphotodiode on the other end of the optical collimator.

In the capacitive sensing part of each sensing pixel, the capacitivesensor 2114 includes a capacitive sensor plate 2116 that iselectromagnetically coupled to a portion of a finger that is eithernearby or in contact with the sensing pixel to perform the capacitivesensing. More specifically, the capacitive sensor plate 2116 and thefinger 2102 interact as two plates of one or more capacitive elements2122 when the finger 2102 is in contact with or substantially near theoptional cover 2104 or a cover on a mobile device that implements thefingerprint sensor device 2100. The number of capacitive sensor plates2116 can vary based on the design of the capacitive sensor 2114. Thecapacitive sensor plate 2116 can be implemented using one or more metallayers. The capacitive sensor plate 2116 is communicatively coupled tocapacitive sensor circuitry 2120 so that the capacitive sensor circuitry2120 can process the signals from the capacitive sensor plate 2116 toobtain data representing the 3D fingerprint structure. A routing orshielding material can be disposed between the capacitive sensor plate2116 and the capacitive sensor circuitry to electrically shield themetal plate 2116. The capacitive sensor circuitry 2120 can becommunicatively coupled to both the capacitive sensor plate 2116 and thephotodetector 2108 to process both the signal from the capacitive sensorplate 2116 and the signal from the photodetector 2108. In FIG. 21A, thecapacitive sensor and the optical sensor within each hybrid sensingpixel are adjacent to and displaced from each other without beingspatially overlapped.

In implementations, the features in the hybrid sensor design in FIG. 21Acan be used in both visible fingerprint sensor package at one end of thedisplay screen structure underneath a top cover glass as shown by theexample in FIG. 1 and invisible fingerprint sensor package under thedisplay screen structure as shown by the example in FIG. 6. For example,the optical fingerprint sensor design in FIG. 6 where the opticalfingerprint sensor underneath the top glass over 50 and the displayscreen structure 54 as illustrated in FIG. 6B may construct each opticaldetector element by an optical collimator element in the detector array123. Therefore, the optical sensing with the optical collimator featurein FIG. 21A may be implemented in a mobile device or an electronicdevice is capable of detecting a fingerprint by optical sensing toinclude a display screen structure; a top transparent layer formed overthe display screen structure as an interface for being touched by a userand for transmitting the light from the display screen structure todisplay images to a user; and an optical sensor module located below thedisplay screen structure to receive light that is returned from the toptransparent layer to detect a fingerprint. The optical sensor moduleincludes an optical sensor array of photodetectors that receive thereturned light and an array of optical collimators to collect thereturned light from the top transparent layer via the display screenstructure and to separate light from different locations in the toptransparent layer while directing the collected returned light throughthe optical collimators to the photodetectors of the optical sensorarray.

FIG. 21B illustrates another example of a fingerprint sensor device 2130that structurally integrates an optical sensor and a capacitive sensorin each hybrid sensor pixel in a spatially overlap configuration in anarray of sensor pixels to reduce the footprint of each hybrid sensingpixel. In some implementations, the hybrid optical-capacitive sensingfingerprint sensor in FIG. 21B may be used in combination with theoptical sensors 41 and 43 by placing it at one end of the display screenstructure 54 underneath a thin top cover glass layer 50 in the visiblefingerprint sensor package as shown in FIG. 1. In other implementations,the optical-capacitive sensing fingerprint sensor in FIG. 21B may beused in combination with the optical sensors 41 and 43 by placing it ina separate sensor structure as such as a button like fingerprint sensorstructure in an opening of the top glass cover 50 in some mobile phonedesigns. The fingerprint sensor device 2130 includes a semiconductorsubstrate 2131, such as silicon. Over the substrate 2131, multiplesensing elements or sensing pixels 2139 are disposed. Each sensingelement or sensing pixel 2139 includes active electronics circuitry area2132 including CMOS switches, amplifier, resistors and capacitors forprocessing sensor signals. Each sensing pixel or sensing element 2139includes a photodetector 2133 disposed or embedded in the activeelectronics circuitry area 2132. A capacitive sensor plate or a topelectrode 2134 of the capacitive sensor for capacitive sensing isdisposed over a photodetector 2133 and includes a hole or opening 2138on the sensor plate 2134 to function also as a collimator of light fordirecting light onto the photodetector 2133. A via 2135 filled withconductive material is disposed to electrically connect the topelectrode 2134 to the active circuit elements 2132. By adjusting theopening or the hole and the distance of the top electrode 2134 with thephotodetector 2133, the light collecting angle 2137 of the photodetector(e.g., photodiode) 2133 can be adjusted. The fingerprint sensor device2130 is covered by a protective cover 2136, which includes hardmaterials, such as sapphire, glass etc. Photodetector 2133 lightcollection angle 2137 can be designed to preserve the spatial resolutionof the image collected by the photodiode arrays. A light source 2140,such as an LED, is placed under the cover, on the side of fingerprintsensor device 2130 to emit light, which is reflected off the finger anddirected toward the photodetector 2133 to capture the fingerprint image.When a finger touches or comes substantially near the protective cover,the finger and the sensing top electrode 2134 in combination form acapacitive coupling (e.g., capacitor 2142) between the human body andsensing top electrode 2134. The fingerprint sensor device 2130 thatincludes both optical and capacitive sensors can acquire images of botha light reflection image of fingerprint and also a capacitive couplingimage. The sensing top electrode 2134 serves dual purpose: 1) forcapacitive sensing, and 2) as a collimator (by fabricating one or moreholes on the sensing top electrode 2134) to direct, narrow or focusreflected light from the finger toward the photodetector 2133. Reusingthe sensing top electrode 2134 eliminates the need for additional metallayer or optical fiber bundle, and thus reduces each pixel size andaccordingly the overall size of the fingerprint sensor device 2130.

In FIG. 21B, the optical sensing design uses the holes or openings 2138formed between the top layer 2136 and the bottom array of photodetectors2133 as optical collimators to select only light rays within certainangles 2137 to preserve the spatial resolution of the image collected bythe photodetectors 2133 in the photodetector array as illustrated.Similar to the fiber or other tubular shaped optical collimators in FIG.21A, the holes or openings 2138 formed between the top layer 2136 andthe bottom array of photodetectors 2133 constitute optical collimatorsto collect the returned light from the top transparent layer via thedisplay screen structure and to separate light from different locationsin the top transparent layer while directing the collected returnedlight through the optical collimators to the photodetectors 2133.

FIG. 22 is a top-down view of an exemplary hybrid fingerprint sensordevice 2200 incorporating both an optical sensor and a capacitive sensorin each hybrid sensing pixel. The fingerprint sensor device 2200 isimplemented as a CMOS silicon chip 2221 that includes an array of hybrid(incorporating both an optical sensor and a capacitive sensor) sensingelements or pixels 2222. Alternatively, the layout in FIG. 22 can alsobe for all optical sensing designs disclosed in this document where theopenings or holes 2223 represent the optical collimators in FIG. 21A or21B. The size or dimension of the sensing elements can be in the rangeof 25 μm to 250 μm, for example. The hybrid sensor device 2200 caninclude an array of support circuitry including amplifiers, ADCs, andbuffer memory in a side region 2224. In addition, the hybrid sensordevice 2200 can include an area for wire bonding or bump bonding 2225. Atop layer 2226 of the hybrid sensor element 2222 can include a metalelectrode for capacitive sensing. One or more openings or holes 2223 canbe fabricated on each top metal electrode 23 to structurally serve as acollimator for directing light in a vertical direction to shine on aphotodetector under the top electrode. Thus, the top layer 2226structure can serve dual purposes of optical and capacitive sensing. Asensor device processor can be provided to process the pixel outputsignals from hybrid sensing pixels to extract the fingerprintinformation.

In addition to sharing the same structure for capacitive sensing and forfocusing light in the vertical direction as a collimator, one instanceof sensor signal detection circuitry can be shared between the opticaland capacitive sensors to detect the sensor signals from both aphotodetector and a capacitive sensor plate.

FIG. 23A illustrates a circuit diagram for an exemplary hybridfingerprint sensing element or pixel 2300 having both capacitive sensingand optical sensing functions for fingerprints. The exemplary sensorpixel 2300 includes sensor signal detection circuitry 2316 toselectively switch between detecting or acquiring sensor signals from asensing top electrode (e.g., a top metal layer) 2308 based on capacitivesensing and a photodetector (e.g., a photodiode) 2314 based on opticalsensing to acquire both a reflective optical image from thephotodetector 2314 and a capacitive coupled image from the capacitivesensor electrode 2308 from a finger. In some implementations, the twoimages from the two sensing mechanisms in each hybrid sensing pixel canbe serially processed by the sensor signal detection circuitry. In theillustrated example, switches 2310 and 2312 have first terminals thatare electrically coupled to the sensing top electrode 2308 and thephotodetector 2314, respectively, and second terminals that are coupledto a common input terminal of the sensor signal detection circuitry 2316to provide corresponding optical detector signal from the photodetector2314 and the corresponding capacitive sensing signal from the sensingtop electrode 2308 to the sensor signal detection circuitry 2316. Whenthe switch 2310 is turned off (CAP_EN=0) and the switch 2312 is turnedon (Optical_en=1), the sensor signal detection circuitry 2316 acquiresthe optical detector signal representing the optical image of thescanned fingerprint received at the particular hybrid sensing pixel. Thesensor signal detection circuitry 2316 can acquire the capacitivesensing signal representing the capacitive image of the scannedfingerprint when switch 2310 CAP_EN=1 and Optical_en=0. After both theoptical and capacitive images are acquired, both images can be processedin downstream circuitry separately and in combination to identify thefingerprint characteristics.

With the two modality of imaging by the above hybrid sensing pixels, theperformance of the fingerprint identification can be enhanced by makinguse of the two types of the images in different ways. This enhancedfingerprint identification can be achieved by the sensor deviceprocessor, such as sensor device processor 2321, for processing thepixel output signals from the hybrid sensing pixels to extract thefingerprint information. For example, the capacitive image can provide a3D image on the depth of the ridges and valleys of the fingerprintfeatures. Complementing the 3D capacitive image, the optical image canprovide a high resolution 2D information on the fingerprintcharacteristics. The optical 2D image having a higher spatial resolutioncan be used to recover the capacitive sensing image resolution becauseboth images information on the same ridges of the fingerprint. In someimplementations where the capacitive sensing method may be moresensitive and accurate on identifying the valleys of the fingerprintthan the optical sensing method, the spatial resolution of imagesacquired using the capacitive sensing method can degrade based on thethickness of the cover. This aspect of the capacitive sensing can besupplemented by the optical sensing. In operation, the sensor responsemay be fixed and the point spread function of the capacitive sensor maybe fixed for all sensor positions. The higher resolution optical sensingcan be used as a resolution recovery method and can be applied on thecapacitive sensing image to enhance the 3D image. A partial highresolution image from optical sensing can be available to help with therecovering method. Thus, the 3D capacitive image can be enhanced toprovide more information on the valleys and ridges by interpolating orrecovering based on the high resolution 2D image.

The enhanced 3D image can provide an improved fingerprint recognitionand matching. In another example, the optical and capacitive images canbe stored together to provide two comparisons each time a fingerprintrecognition or matching is performed. The use of two types of images forcomparison enhances the accuracy and security of the fingerprint sensingsystem.

The sensor signal detection circuitry 2316 can be implemented in variousways using a number different circuitry designs. In one example,integrator sensing circuitry 2318 can be implemented to store theelectric charges caused by ridges and valleys touching or beingsubstantially near the cover of the fingerprint sensor device of thecover of the mobile device. The inclusion of the integrator circuitry2318 enhances the signal-to-noise ratio (SNR). The integrator sensingcircuitry includes an operational amplifier 2322 to amplify a sensorsignal, such as a capacitance related or optical related signal (e.g.,voltage signal), detected by the sensing top electrode 2308 or thephotodetector 2314 of the exemplary sensor pixel 2300. The sensing topelectrode 2308 that include a conductive material, such as one of avariety of metals is electrically connected to a negative or invertingterminal 2328 of the amplifier 2322 through the switch 2310. The sensingtop electrode 2108 and a local surface of the finger 2302 function asopposing plates of a capacitor Cf 2302. The capacitance of the capacitorCf 2302 varies based on a distance ‘d’ between the local surface of thefinger and the sensing top electrode 2308, the distance between the twoplates of the capacitor Cf 2302. The capacitance of capacitor Cf 2302 isinversely proportional to the distance ‘d’ between the two plates of thecapacitor Cf 2302. The capacitance of capacitor Cf 2302 is larger whenthe sensing top electrode 2308 is opposite a ridge of the finger thanwhen opposite a valley of the finger.

In addition, various parasitic or other capacitors can be formed betweendifferent conductive elements in the exemplary sensor pixel 2300. Forexample, a parasitic capacitor CP 2304 can form between the sensing topelectrode 2308 and a device ground terminal 2305. Device ground iscoupled to earth ground closely. Another capacitor Cr 2324 can formbetween an output conductor of the amplifier 2322 and the negative orinverting terminal 2328 of the amplifier 2322 and functions as afeedback capacitor to the amplifier 2322. Also, a switch 2326 can becoupled between the output of the amplifier 2322 and the negative orinverting terminal 2328 of the amplifier 2322 to reset the integratorcircuitry 2318.

The positive terminal of the amplifier 2322 is electrically connected toan excitation signal Vref. The excitation signal Vref can be directlyprovided to the positive terminal of a dedicated amplifier in eachsensor pixel. By providing the excitation signal Vref directly to thepositive terminal of the amplifier 2322, the exemplary sensor pixel 2100becomes an active sensor pixel. In addition, providing the excitationsignal Vref directly to the positive terminal of the amplifier 2322eliminates the need to include an excitation electrode, common to allsensor pixels, which reduces a conductive (e.g., metal) layer from thesemiconductor structure of the sensor chip. In some implementations, anoptional excitation electrode 2306 can be implemented to enhance the SNRbased on the design of the sensor pixel. In addition, by providing theexcitation signal Vref 2330 directly to the amplifier 2322, theexcitation signal Vref 2322 is not applied directly to the finger toavoid potentially irritating or injuring the finger. Moreover, when theexcitation electrode for applying the excitation signal directly to thefinger is not used, all components of the fingerprint sensor device canbe integrated into a single packaged device, and the entire fingerprintsensor device can be disposed under the protective cover glass. With theentire fingerprint sensor device disposed under the protective coverglass, the fingerprint sensor device is protected from the finger andother external elements that can potentially damage the fingerprintsensor.

In FIG. 23A, the output signal (optical and capacitive) of the sensorsignal detection circuitry 2316 (e.g., Vpo of the amplifiers 2322) inthe sensor pixels 2300 is electrically coupled to a switch 2320 toselectively output the output signal Vpo from the sensor pixel 2300 to asignal processing circuitry including a filter. The switch 2320 can beimplemented using a transistor or other switching mechanisms andelectrically coupled to a controller to control the switching of theswitch 2320. By controlling the switches 2320, 2310 and 2312, the sensorpixels in an array of sensor pixels can be selectively switched betweenacquiring the optical signals and the capacitive signals. In oneimplementation, the optical or capacitive signal can be acquired foreach line, row or column of sensor pixels in the array and then switchedto acquire the other type of signal for the line, row or column. Theswitching between the optical and capacitive signal acquisition can beperformed line-by-line. In another implementation, one type of signal(capacitive or optical) can be acquired for all sensor pixels orelements in the array and then switched to acquire the other type ofsignal for all of the sensor pixels or elements. Thus, the switchingbetween acquisition of different signal types can occur for the entirearray. Other variations of switching between acquisition of the twotypes of sensor signals can be implemented.

FIG. 23B illustrates a circuit diagram for another exemplary hybridfingerprint sensing element or pixel 2340. The hybrid fingerprintsensing element or pixel 2340 is substantially the same as the hybridfingerprint sensing element or pixel 2300 with respect to the componentshaving the same reference number. For descriptions of the commoncomponents having the same reference number, refer to the description ofFIG. 23A.

The hybrid fingerprint sensing element or pixel 2340 implements thesensing top electrode 2308 to include a hole or opening 2342 thatfunctions as a collimator to focus or narrow the reflected light 2344toward the photodetector 2314 (e.g., photodiode). The photodetector 2314can be positioned or disposed below the collimator implemented using thesensing top electrode 2308 to capture the reflected light 2344 focusedby the collimator 2308.

In some implementations, separate instances of sensor signal detectioncircuitry can be included for the optical and capacitive sensors todetect in parallel the sensor signals from both a photodetector and acapacitive sensor plate.

FIG. 23C illustrates a circuit diagram of an exemplary hybridfingerprint sensing element or pixel 2350 for performing paralleldetection of sensor signals from the photodetector and the capacitivesensor plate. The hybrid fingerprint sensing element or pixel 2350 issubstantially the same as the hybrid fingerprint sensing element orpixel 2340 with respect to the components having the same referencenumber. For descriptions of the common components having the samereference number, refer to the description of FIG. 23A.

To perform sensor signal detection from both the capacitive plate andthe photodetector in parallel, the hybrid fingerprint sensing element orpixel 2350 includes separate sensor signal detection circuitry 2316 and2317 communicatively coupled to the sensing top electrode 2308 and thephotodetector 2324 respectively. Sensor signal detection circuitry 2317can be implemented to be substantially similar to sensor signaldetection circuitry 2316. In some implementations, switches 2310 and2312 can be disposed to have first terminals that are electricallycoupled to the sensing top electrode 2308 and the photodetector 2314,respectively, and second terminals that are coupled to respective sensorsignal detection circuitry 2316 and 2317 to provide the optical detectorsignal from the photodetector 2314 and the capacitive sensing signalfrom the sensing top electrode 2308 to the sensor signal detectioncircuitry 2316 and 2317 respectively When the switches 2310 and 2312 areturned on and off together, the sensor signal detection circuitry 2316and 2317 can perform sensor signal detection from the capacitive plate2308 and the photodetector 2314 in parallel. When the switches 2310 and2312 are turned on and off out of phase with each other, the sensorsignal detection circuitry 2316 and 2317 can perform sensor signaldetection from the capacitive plate 2308 and the photodetector 2314 inseries. In addition, the sensor device processor 2321 can becommunicatively coupled to the sensor signal detection circuitry 2316and 2317 either directly or indirectly through switches 2320A and 2320Bto process the detected sensor signals from the capacitive plate 2308and the photodetector 2314 in parallel or in series.

In another aspect of the disclosed technology, the optical sensordescribed with respect to FIGS. 21A, 21B, 22, 23A and 23B can be used tomeasure human heart beat by measuring the reflected light intensitychange with time caused by blood flow variations in fingers due to theheart beat and pumping actions of the heart. This information iscontained in the received light that is reflected, scattered or diffusedby the finger and is carried by the optical detector signal. Thus, theoptical sensor can serve multiple functions including acquiring anoptical image of the fingerprint and to measure human heart beat. Inimplementations, a sensor device processor is used to process one ormore optical detector signals to extract the heart beat information.This sensor device processor may be the same sensor device processorthat processes the pixel output signals from optical sensing pixels orhybrid sensing pixels to extract the fingerprint information.

FIGS. 24A, 24B, 24C and 24D show process flow diagrams of an exemplaryprocess 2400, for performing fingerprint sensing by a hybrid fingerprintsensor that incorporates optical and capacitive sensors. A method 2400formed by a fingerprint sensor device includes detecting, by an array ofsensor pixel circuitry in the fingerprint sensor device, capacitanceassociated with a touch from a finger indicative of a fingerprint scan(2402). The method includes detecting, by the array of sensor pixelcircuitry in the fingerprint sensor device, an optical signal associatedwith light reflected from the finger (2404). The method includesoutputting, by the array of sensor pixel circuitry, output signalsresponsive to the detected capacitance and optical signal (2406). Theoutput signals can be processed to perform fingerprint identification(2408). Processing the output signals (2408) can include processing theoutput signals to acquire an optical image of the fingerprint and acapacitive image of the fingerprint (2410). The acquired optical imageand capacitive image can be stored as two types of registeredfingerprint images for comparing scanned fingerprints during fingerprintidentification (2412). Processing the output signals (2408) can includeusing the optical image to recover information on the ridges of thefingerprint in the capacitive image (2414). The output signals can beintegrated for the array of sensor pixel circuitry (2416). Integratingcan include integrating by all of the sensor pixel circuitry in thearray the output signals in parallel.

As mentioned above, in some implementations of the optical sensing offingerprints and other sensing operations, the optical sensor module maybe packaged in a discrete device configuration in which the opticalsensor module is a distinct structure from the display screen and has astructural border or demarcation with the display screen, e.g., a buttonlike fingerprint sensor structure in an opening of the top glass coverin some mobile phone designs. FIGS. 25A and 25B illustrate an example ofsuch a design for a mobile phone, a tablet or other devices where theoptical sensor module is outside the display screen assembly 10 as aseparate structure. The top cover glass layer 50 is fabricated toinclude an opening or through hole 2510 to accommodate for at least thetop structure of the optical sensor module. This opening or through hole2510 can be in a desirable shape such as a circular shape like a button,a rectangular shape such as the elongated rectangular shape shown inFIG. 25A, or other suitable shapes.

The optical sensor module is shown in a cross-sectional view along lineBB′ in FIG. 25A in FIG. 25B. Disposed above the optical detector array23 are receiving optics 24 (e.g., optical lenses or opticalcollimators), an optical transparent layer or space 31 above the optics24 and the top optical transparent layer 2520 that forms an opticalinput interface for fingerprint sensing and other optical sensing. Aback board 25 with integrated circuitry is disposed below the detectorarray 23. In some implementations, the detector array 23 can beintegrated into the backboard 25. The backboard 25 can be disposed overa flexible printed circuit (FPC) 27.

Referring to FIG. 25A, the detector array 23 includes multiple detectorelements arranged in different detector zones including a fingerprintdetector zone 33 located in the central area of the detector array 23for fingerprint and fingerprint property detection and one or moreadditional detector zones 41 and 43 for other optical sensing functions,where the additional detector zones 41 and 43, as illustrated in FIG.1C, may be environment and blood flow detector zones 41 and 43 forenvironment and blood flow speed detection. In addition, referencenumbers 37 and 39 represent optical illumination zones to producedesired illumination for the additional optical sensing, e.g., emittingred light or light of a desired spectral range to illuminate a user'sfinger for sensing blood flow speed or glucose level.

The disclosed technology can be applied to implement the fingerprintsensor in smartphones, tablets, laptops, portable game machines,portable controllers, and other electronic devices that uses secureaccess. In a device based on the disclosed technology, a controlcircuit, which may include a control processor, can be used to providethe control operations disclosed, e.g., the modulation of theillumination light in optical fingerprint sensing and other controlfunctions and operations.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a sub combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

What is claimed is:
 1. An electronic device with touching sensing andfingerprint sensing capabilities, comprising: a touch screen thatprovides touch sensing operations; a top transparent layer formed overthe touch screen as an interface being touched by a user for the touchsensing operations; and an optical sensor module located below the touchscreen to receive light that is returned from the top transparent layerand transmits through the touch screen, the optical sensor moduleincluding an optical detector array of photodetectors positioned toreceive at least a portion of the returned light to detect afingerprint.
 2. The electronic device as in claim 1, wherein: theoptical sensor module includes an array of optical collimators locatedbetween the touch screen and the optical detector array to direct thereceived portion of the returned light to the photodetectors through theoptical collimators.
 3. The electronic device as in claim 2, wherein:the optical collimators include optical fibers.
 4. The electronic deviceas in claim 2, wherein: the optical collimators include a layer and anarray of holes through the layer.
 5. The electronic device as in claim1, wherein: the touch screen includes display pixels that produce lightfor displaying images and for illuminating a user's finger in touch withthe top transparent layer to produce the returned light.
 6. Theelectronic device as in claim 5, further comprising: one or moreillumination light sources that are separate from the display pixels ofthe touch screen and produce illumination light for illuminating auser's finger in touch with the top transparent layer to produce thereturned light.
 7. The electronic device as in claim 1, wherein: thetouch screen includes a fingerprint sensing zone for a user to touch forfingerprint sensing to generate the returned light received by theoptical detector array for detecting a fingerprint; a first opticalsensing zone and a second optical sensing zone; the optical moduleincludes (1) a first additional optical detector located on a first sideof the optical detector array to receive a portion of the returned lightfrom the first optical sensing zone, and (2) a second additional opticaldetector located on a second opposite side of the optical detector arrayto receive a portion of the returned light from the second opticalsensing zone, wherein the first and second additional optical detectorsproduce detector signals indicating whether the returned light isreflected from a finger of a live person.
 8. The electronic device as inclaim 7, wherein: the first and second additional optical detectorsproduce detector signals indicating a heartbeat of a live person.
 9. Theelectronic device as in claim 7, wherein: the first and secondadditional optical detectors produce detector signals indicating a bloodflow speed of a live person.
 10. The electronic device as in claim 7,wherein: the optical sensor module includes an array of opticalcollimators located between the touch screen and the optical detectorarray to direct the received portion of the returned light to thephotodetectors through the optical collimators.
 11. The electronicdevice as in claim 1, wherein: the touch screen includes display pixelsthat produce light for displaying images and for illuminating a user'sfinger in touch with the top transparent layer to produce the returnedlight; and the touch screen includes a fingerprint sensing zone for auser to touch for fingerprint sensing to receive illumination light fromone or more display pixels that are located to generate illuminationlight to the fingerprint sensing zone in a way that the illuminationlight undergoes total optical reflection at a top surface of the toptransparent layer to direct the totally reflected light to the opticaldetector array for detecting a fingerprint.
 12. The electronic device asin claim 1, comprising: a control that controls illumination light infingerprint sensing by the optical sensor module.
 13. The electronicdevice as in claim 12, wherein the control modulates the illuminationlight in fingerprint sending.